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Abstract 

An  improved  method  for  deformable  shape-based  image 
segmentation  is  described.  Image  regions  are  merged  to¬ 
gether  and/or  split  apart,  based  on  their  agreement  with 
an  a  priori  distribution  on  the  global  deformation  param¬ 
eters  for  a  shape  template.  Perceptually -motivated  criteria 
are  used  to  determine  where/how  to  split  regions,  based  on 
the  local  shape  properties  of  the  region  group ’s  bounding 
contour.  A  globally  consistent  interpretation  is  determined 
in  part  by  the  minimum  description  length  principle.  Ex¬ 
periments  show  that  model-guided  split  and  merge  yields  a 
significant  improvement  in  segmention  over  a  method  that 
uses  merging  alone. 

1  Introduction 

Retrieval  by  shape  is  a  key  topic  in  content-based  image 
retrieval  research.  Unfortunately,  retrieval  by  shape  re¬ 
quires  object  detection  and  segmentation.  In  [16,  23]  we 
described  a  system  for  automatic  shape-based  indexing  of 
image  databases  that  employs  a  shape  model  to  guide  re¬ 
gion  grouping.  A  limitation  of  the  system  was  that  it  em¬ 
ployed  model-based  region  grouping  only,  without  region 
splitting.  However,  it  is  well-known  that  split/merge  in  re¬ 
gion  segmentation  [11]  tends  to  yield  improved  accuracy 
over  methods  that  employ  only  splitting  or  only  merging. 

In  this  paper,  a  method  is  proposed  for  incorporating  a 
model  splitting  step  in  the  previous  model-guided  region 
grouping  algorithm  [16,  23].  Candidate  region  splits  are 
determined  using  criteria  that  are  motivated  by  human  psy¬ 
chophysics.  In  addition,  cues  obtained  from  a  statistical 
shape  model  are  used  to  prioritize  the  selection  of  candi¬ 
date  region  splits.  The  resulting  algorithm  combines  bot¬ 
tom  up  processing  with  top-down  (model-guided)  process¬ 
ing.  As  will  be  seen  in  the  experiments,  the  split  and  merge 
approach  yields  a  significant  improvement  over  the  previ¬ 
ous  algorithm  in  detection  and  segmentation  of  deformable 
shapes.  The  method  can  detect  multiple  shapes  even  in  the 
presence  of  poor  initial  segmentation,  shadows,  or  when 
shapes  in  the  image  touch. 

*  This  work  was  supported  in  part  through  Office  of  Naval  Research 
Young  Investigator  Award  N000 14-96- 1-0661,  and  National  Science 
Foundation  grants  IIS-9624168  and  EIA-9623865. 


2  Related  work 

Some  traditional  methods  for  region  splitting  include  sep¬ 
aration  of  touching  features  by  erosion/dilation,  water¬ 
shed  segmentation,  etc.  [22].  However,  morphological 
operation-based  methods  only  deal  with  regular  shapes, 
or  where  there  are  only  bottleneck  connections  between 
touching  objects. 

A  natural  question  is:  How  does  the  human  vision  sys¬ 
tem  parse  a  region  into  parts?  Perhaps  such  strategies  can 
be  adapted  to  solve  our  problem.  Some  theorists  postulate 
that  there  is  a  set  of  basic  shape  primitives  [3,  4,  18,  19] 
that  are  useful  in  finding  parts  and  describing  them.  Others 
postulate  that  there  are  rules,  based  on  geometric  properties 
alone,  by  which  humans  perceive  boundaries  between  parts 
for  a  given  shape  [10,  24].  Still  others  propose  theories 
based  on  the  relationship  between  the  nonlinear  diffusion 
equation  and  shape  perception  [14,  28].  We  will  make  use 
of  both  shape  constraints  (deformable  template  and  statis¬ 
tical  priors)  and  geometric  properties  of  the  region  contour 
(curvature  minima)  in  guiding  splits. 

In  related  work,  Hoffman  and  Richards  [10,  20]  pro¬ 
vided  the  minima  rule  for  shape  parsing:  cut  each  silhou¬ 
ette  into  parts  at  concave  cusps  and  negative  minima  of 
curvature.  For  a  2D  silhouette,  the  minima  rule  provides 
boundary  points  on  the  silhouette  outline,  and  part  cuts 
must  pass  through  them.  However,  the  minima  rule  does 
not  define  the  part  cuts  themselves  -  it  only  constrains  them 
to  pass  through  the  boundary  points  it  provides. 

There  are  perceptual  constraints  beyond  the  minima  rule 
that  affect  one’s  parsing  preference  [24].  Part  salience  [9] 
and  the  short-cut  rule  [25]  were  proposed  to  embody  those 
constraints.  Hoffman  and  Singh [9]  isolated  three  factors  in 
part  salience:  relative  area,  amount  of  protrusion,  and  nor¬ 
malized  curvature  across  the  part  boundary.  However,  they 
do  not  integrate  these  into  a  shape  partitioning  scheme. 
The  short-cut  rule  [25]  is  simple:  if  boundary  points  can  be 
joined  in  more  than  one  way  to  parse  a  silhouette,  choose 
the  parsing  that  uses  the  shortest  cuts.  A  cut  is  defined  to 
be  (1)  a  line  that  (2)  crosses  an  axis  of  local  symmetry,  (3) 
joins  two  points  on  the  silhouette,  such  that  (4)  at  least  one 
of  the  two  points  has  negative  curvature. 

The  short-cut  rule  has  some  limitations.  First,  the  length 
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of  the  cut  only  involves  minimal  shape  information.  Sec¬ 
ond,  restricting  the  cut  to  cross  a  symmetry  axis  is  prob¬ 
lematic,  because  robust  computation  of  symmetry  axes  is 
difficult  in  general.  To  circumvent  these  problems,  [21] 
proposed  segmenting  regions  into  roughly  convex  parts  via 
an  optimization  procedure  at  multiple  scales.  Since  the 
method  uses  exhaustive  search  to  find  the  optimal  cuts,  the 
algorithm  is  slow  despite  the  use  of  a  multi-scale  approach. 
In  addition,  the  method  requires  that  the  number  of  cuts  for 
the  shape  partitioning  is  known. 

Latecki,  et  al.,  [15]  used  a  contour  evolution  method  to 
identify  convex  parts  at  different  stages.  The  assumption  is 
that  significant  visual  parts  will  become  convex  at  higher 
stages  of  the  evolution.  In  related  work,  Belyaev,  et  al.[l] 
developed  a  polygonal  curve  evolution  method  that  can  be 
used  for  multi-scale  shape  analysis.  A  major  drawback  of 
contour  evolution  approaches  is  that  it  is  unclear  how  to 
distinguish  cuts  between  objects  from  cuts  between  parts 
inside  the  same  object. 

Two-dimensional  part  decomposition  methods  to  date 
have  had  several  drawbacks.  There  is  no  cut  rule  that  is 
good  for  all  cases.  However,  our  problem  is  different  in 
that  we  can  make  use  of  a  deformable  template  model  in 
determining  cuts  employed  in  region  splitting.  Use  of  the 
shape  model,  which  embodies  knowledge  about  a  partic¬ 
ular  class  of  objects  and  the  prior  distribution  on  nonrigid 
deformation,  will  improve  cut  selection  considerably. 

3  Review  of  Basic  Approach 

In  [16,  23]  we  proposed  a  method  that  uses  a  deformable 
model  to  guide  grouping  of  image  regions.  Our  basic  ap¬ 
proach  extends  methods  employed  in  active  contours  [13] 
and  deformable  templates  [7,  8,  12,  17,  27]  to  the  problem 
of  automatic  object  detection  and  segmentation  via  region 
grouping.  We  will  now  briefly  review  our  basic  approach. 
For  more  detailed  discussion  of  our  approach,  including 
parameter  settings,  please  see  [23]. 

An  overview  of  the  approach  is  shown  in  Fig.  1.  In  the 
pre-processing  stage,  the  input  image  is  over-segmented 
via  standard  region  merging  algorithms  [2,  6].  The  out¬ 
put  of  this  module  includes  a  standard  region  adjacency 
graph.  An  edge  map  is  also  computed;  notable  edges  and 
their  strengths  are  detected  via  standard  image  processing 
methods.  The  resulting  edge  map  will  be  used  to  constrain 
consideration  of  possible  grouping  hypotheses  later  in  re¬ 
gion  merging. 

The  system  then  tests  various  combinations  of  candidate 
region  groupings  to  obtain  an  optimal  labeling  of  the  im¬ 
age.  The  shape  model  is  deformed  to  match  each  grouping 
hypothesis  g i  in  such  a  way  as  to  minimize  a  cost  function: 

=  OtEcolor  (1  Cf)((l  —  (3)Earea  +  fiEdeform), 

(1) 

where  a  and  ft  are  scalar  constants  with  values  in  the  range 
[0, 1]  that  control  the  relative  importance  of  the  three  terms: 


Figure  1:  Diagram  of  the  deformable  template-based  region 
merging  system.  An  input  image  (image  of  bananas)  under¬ 
goes  pre-processing,  which  results  in  an  over- segmentation  and 
an  edge  map.  These  are  inputs  to  the  model-based  region  group¬ 
ing  stage  (using  a  deformable  banana  template).  The  final  output 
includes  region  groupings  (shown  in  top  output  image)  for  de¬ 
tected  objects  (four  bananas),  and  recovered  models  for  the  ob¬ 
jects  (shown  in  bottom  output  image). 

E coior  is  a  region  color  compatibility  term  for  the  region 
grouping,  Earea  is  a  region/model  area  overlap  term,  and 
Edeform  is  a  deformation  energy  for  the  shape  model.  A 
model  fitting  procedure  is  used  to  compute  the  cost  in  Eq.  1 
via  the  downhill  simplex  method. 

A  deformation  term  enforces  a  priori  constraints  on  the 
amounts  and  types  of  deformations  allowed  for  the  tem¬ 
plate;  i.e.: 

Edef  orm  oc-logP(a|fi),  (2) 

where  P(a|f2)  gives  the  prior  distribution  on  global  defor- 
mation  parameters,  a,  for  a  particular  shape  class  U.  In  our 
experience,  the  use  of  a  Gaussian  model  for  the  prior  distri¬ 
bution  on  global  deformation  leads  to  reliable  shape-based 
image  segmentation.  An  estimate  of  the  prior  distribution 
is  computed  in  a  supervised  fashion,  for  a  given  set  of  train¬ 
ing  examples  for  that  shape  class.  In  our  implementation, 
linear  and  quadratic  polynomials  are  used  to  model  defor¬ 
mation  due  to  stretching,  shearing,  bending,  and  tapering. 

Further,  in  order  to  test  the  quality  of  a  possible  parti¬ 
tioning,  a  global  cost  function  for  partitioning  the  whole 
image  is  defined: 

n 

£  =  (i  -7)X!riE(gi) +T,n’ 

2=1 

where  7  is  a  constant  factor,  n  is  the  number  of  the  group¬ 
ings  in  the  current  image  partitioning,  r\  is  the  ratio  of  ith 
group  area  to  the  total  area  of  connected  regions,  and  E(g  i) 
is  the  cost  function  for  the  group  g i  (Eq.  1).  The  highest 
confidence  first  (HCF)  algorithm  [5]  is  used  to  find  an  ap¬ 
proximately  optimal  value  for  Eq.  3.  The  approximately 


Figure  2:  Error  in  segmentation  The  input  image  is  a  blood  cell 
micrograph  (a).  The  model-guided  segmentation  is  shown  in  (b), 
with  cell  regions  (grey  areas)  and  fitting  models  (black  contours). 


Figure  3:  Detected  cut  points  (a)  and  selected  cut  (b)  for  Fig.  2. 
Dotted  contours  in  (a)  are  the  model  boundaries,  dark  square 
marks  represents  the  candidate  cut  points. 


optimal  region  groupings  obtained  via  HCF  along  with  re¬ 
covered  shape  models  are  shown  in  Fig.  1. 

4  Model-Guided  Region  Splitting 

As  observed  in  experiments  [16],  our  model-guided  re¬ 
gion  merging  scheme  is  often  able  to  obtain  a  satisfactory 
segmentation  despite  clutter,  shadows,  variation  in  illumi- 
nant,  and  shape  deformation.  However,  performance  of  the 
method  is  hindered  by  its  assumption  of  an  over- segmented 
image  to  be  used  as  input  to  the  shape-based  region  group¬ 
ing  subsystem.  In  general,  it  is  not  possible  to  consistently 
over- segment  all  input  images.  Invariably,  some  parts  of 
images  are  under- segmented,  which  means  that  some  of 
the  initial  regions  correspond  with  parts  of  more  than  one 
object  in  the  scene.  An  example  error  in  segmentation  of  a 
blood  cell  micrograph  is  shown  in  Fig.  2.  In  addition,  the 
HCF  optimization  procedure  sometimes  becomes  trapped 
in  a  local  minimum  that  yields  an  incorrect  region  merging. 

To  address  these  problems,  a  region  splitting  step  must 
be  added  to  our  system.  Perceptually-motivated  criteria 
will  be  used  to  determine  where/how  to  split  a  region, 
based  on  the  local  shape  properties  of  the  region’s  bound¬ 
ing  contour.  We  will  also  make  use  of  our  deformable 
shape-model  in  guiding  the  selection  of  splits.  In  consider¬ 
ing  a  split,  the  change  in  the  fitting  cost  (Eq.  3)  after  split¬ 
ting  provides  a  measure  for  the  quality  of  the  cut. 

4.1  Region  Splitting  Criteria 

In  order  to  detect  candidate  regions  or  groups  for  splitting, 
the  following  test  can  be  utilized: 

1.  At  the  beginning  of  the  model-based  region  grouping 
stage,  check  each  region.  If  that  region’s  model  fitting 
cost  (Eq.  1)  is  larger  than  a  threshold,  then  it  is  a  split¬ 
ting  candidate. 

2.  At  the  end  of  merging  via  HCF,  for  each  group,  if  its 
model  fitting  cost  is  larger  than  the  threshold,  then  it  is  a 
splitting  candidate. 

The  split  threshold  can  be  obtained  through  statistical  anal¬ 
ysis  for  the  training  examples.  Since  the  number  of  regions 
in  the  over- segmentation  result  is  large  typically,  we  only 
utilize  the  splitting  operation  after  the  model-based  HCF 


merging  stage  (for  computational  efficiency).  If  new  re¬ 
gions  occur  in  the  model  based  splitting  stage,  then  the 
model-based  merging  stage  is  invoked  again  to  refine  the 
final  result.  Eq.  3  is  used  both  in  splitting  and  merging  for 
evaluating  the  feasibility  of  split/merge  operations,  enforc¬ 
ing  the  Minimum  Description  Length  (MDL)  principle. 

4.2  Selecting  Candidates  of  Cut  Endpoints 

If  a  group  of  regions  is  a  splitting  candidate,  then  we  use 
perceptual  criteria  and  model  fitting  information  to  select 
the  splitting  boundaries.  After  we  have  determined  the 
overlapping  between  the  fitted  model  and  the  underlying 
region  for  splitting,  as  shown  in  Fig.  2(b),  we  can  get  the 
curve  segments  of  the  region  boundary  inside  the  model 
area  (the  thick  gray  curve  segments  shown  in  Fig.  3(a)). 
The  search  for  candidates  of  cut  endpoints  will  be  limited 
to  these  segments. 

One  criterion  of  splitting  methods  is  robustness:  the 
method  should  be  insensitive  to  small  changes  of  shape  due 
to  differences  among  the  members  of  a  class  or  to  small  dif¬ 
ferences  in  view  position.  Because  curvature  computation 
is  more  complex  and  sensitive  to  noise,  we  do  not  utilize 
the  minima  rule  for  the  region  splitting,  instead  we  will 
search  for  cut  endpoints  in  the  concave  segments  on  the  re¬ 
gion  boundary  along  lines  of  [26].  Cut  endpoints  are  gen¬ 
erally  located  in  the  region  boundary’s  concave  segments, 
which  are  most  likely  inside  the  fitted  model.  Making  use 
of  the  difference  between  the  region  boundary  and  model 
boundary  can  make  the  splitting  algorithm  more  robust  and 
more  efficient  since  knowledge  about  the  object  can  be  em¬ 
ployed  in  the  splitting.  The  concave  segments  detection 
method  is  as  follows: 

1 .  Intersect  the  region  boundary  with  the  model  boundary. 
This  yields  a  collection  of  curve  segments  s  i  of  the  re¬ 
gion  boundary  that  lie  with  in  the  model  area. 

2.  For  each  curve  segment  Si,  for  s^s  starting  point  to  its 
end  point,  check  the  convexity  of  each  point  along  the 
segment.  In  other  words,  for  each  point,  check  its  small 
neighborhood  to  decide  whether  this  point  is  on  a  convex 
segment  or  concave  segment.  The  neighborhood  size  l  n 


will  influence  the  accuracy  and  robustness  of  the  result. 1 

3.  After  smoothing  the  convexity  of  each  point  to  eliminate 
noise  influence,  the  whole  segment  Si  will  be  decom¬ 
posed  into  concave  and  convex  sub-segments. 

4.  For  each  concave  sub-segment  from  ,s  ,  check  its  length. 
If  the  length  is  compatible  with  the  specified  neighbor¬ 
hood  size  ln  for  checking  convexity,  then  select  the  cen¬ 
ter  point  of  this  sub- segment  as  a  candidate  cut  point. 

In  the  concave  segments  detection  method,  there  is  also 
another  criterion  which  can  be  used  in  selecting  candidate 
points [26]:  If  the  length  of  a  concave  sub-segment  is  com¬ 
parable  with  its  adjacent  convex  sub-segments,  then  regard 
this  convexity  change  as  due  to  the  natural  shape  change  of 
the  object,  and  not  well  suited  for  splitting.  Only  when  the 
length  of  a  concave  sub- segment  is  significantly  smaller 
than  those  of  its  adjacent  convex  sub-segments,  is  a  candi¬ 
date  point  selected  from  it.  However,  in  our  experiments, 
this  criterion  is  too  strict;  real  cut  endpoints  may  be  omit¬ 
ted,  due  to  the  non-smoothness  of  the  region  boundaries. 

One  added  benefit  in  our  approach  is  that  we  can  make 
use  of  overlap  between  the  deformed  model  and  the  under¬ 
lying  region  for  determining  if  and  how  to  split  the  region. 
This  makes  the  splitting  algorithm  more  robust  and  effi¬ 
cient  since  knowledge  about  the  object  can  be  employed  in 
testing  and  selecting  cuts.  It  should  be  noted  that  our  algo¬ 
rithm  also  considers  the  boundaries  of  holes  in  determining 
cut  points,  if  the  holes  are  inside  the  fitted  model. 

The  cut  point  candidates  obtained  will  be  tested  in  the 
cut  selection  stage.  It  is  likely  that  not  all  the  cut  point 
candidates  will  be  used  in  the  final  cut  (maybe  none  at  all). 
Selection  of  cuts  is  described  in  the  next  section. 

4.3  Strategies  for  Selecting  a  Cut 

Given  a  set  of  candidate  cut  end  points  for  splitting  the  re¬ 
gion,  there  are  many  possible  region  cuts  possible.  In  our 
model-based  splitting  method,  the  quality  of  a  cut  can  be 
determined  by  calculating  the  change  in  global  cost  Eq.  3 
that  results  from  that  region  cut.  Put  differently,  if  the  com¬ 
bined  cost  of  the  two  new  regions  is  less  than  the  cost  of 
the  original  region,  then  this  cut  can  be  regarded  as  feasi¬ 
ble.  For  example,  given  region  grouping  g  can  be  split  into 
gi  and  g2  by  a  cut,  we  compare  the  old  cost 

(1  -  7)rE(g)  +  7,  (4) 

and  the  new  cost 

(1 -7)(riE(gi) +r2E(g2)) +  27,  (5) 

where  7  and  r  are  as  defined  in  Eq.  3,  E(g)  in  Eq.  1.  If  the 
new  cost  is  less  than  the  old  cost,  then  the  cut  is  feasible. 

To  select  a  cut  based  on  a  set  of  candidates  of  cut  end¬ 
points,  the  direct  strategy  is:  test  all  possible  cuts.  For 

!We  choose  ln  =  6  in  our  experiments. 


all  possible  cuts,  compare  the  cut  quality  via  Eqs.  4  and  5 
and  select  the  one  with  the  least  cost.  Fig.  3(b)  shows  a 
cut  selected  based  on  this  strategy.  The  drawback  of  ex¬ 
haustive  search  is  the  computation  required.  We  therefore 
tested  two  alternative  strategies  for  selecting  a  cut:  modi¬ 
fied  short-cut  rule  [25],  and  a  queuing  strategy. 

In  the  modified  short-cut  rule ,  the  cut  of  shortest  length 
is  tested  first.  If  this  cut  is  a  feasible  cut,  then  stop.  Oth¬ 
erwise,  delete  the  candidate  point  with  least  significance 
from  this  cut,  and  repeat  to  test  the  shortest  cuts  given  the 
remaining  candidate  points  until  a  feasible  cut  is  found  or 
there  is  no  cut  remaining  to  test. 

In  the  queuing  strategy ,  sort  all  possible  cuts  according 
to  the  cut  length  and  construct  a  queue.  Fetch  cuts  from 
this  queue  until  a  feasible  cut  is  obtained.  In  other  words, 
first  test  the  minimum  cut,  then  the  second  minimum  cut, 
etc. ,  until  a  feasible  cut  is  found  or  no  cuts  remain. 

If  there  is  only  one  candidate  point  detected  in  a  region 
for  splitting,  one  possible  solution  is  to  generate  a  cut  from 
this  candidate  point  to  cross  the  local  symmetry  axis  of  the 
region  [25].  However,  the  robust  computation  of  axes  is 
difficult.  Our  simple  alternative  is  to  fit  a  line  to  the  corre¬ 
sponding  curve  segment  of  the  model  boundary,  and  gen¬ 
erate  a  cut  from  this  candidate  point  to  follow  the  normal 
direction  of  the  fitted  line.  The  motivation  for  this  approxi¬ 
mation  is  that  model  boundaries  tend  to  be  locally  smooth. 

4.4  Smooth  Cuts  from  Model  Boundaries 

There  is  another  benefit  from  computing  the  overlap  be¬ 
tween  the  recovered  model  and  the  underlying  region.  If 
one  object  is  incorrectly  merged  with  a  small  part  of  its  ad¬ 
jacent  object(as  shown  in  Fig.  4(c)  and  (d)),  the  curve  seg¬ 
ments  of  model  boundary  inside  the  region  are  also  good 
candidates  of  cuts  since  the  recovered  model  is  more  sta¬ 
ble  to  noise.  In  Fig.  4(c)  and  (d),  one  curve  segment  of  the 
model  boundary  inside  one  fish  region  separates  the  incor¬ 
rect  merging  part  from  the  fish  body.  Based  on  curve  seg¬ 
ments  of  the  model  boundary  inside  the  region,  a  smooth 
cut  can  be  obtained  instead  of  only  straight  cuts.  There¬ 
fore,  the  precision  of  the  region  boundary  can  be  improved 
as  well.  The  resulting  segmentation  after  split/merge  step 
is  shown  in  Fig.  4(e)  and  (f). 

To  reduce  the  computation  complexity,  smooth  cuts  are 
only  considered  if  there  is  no  feasible  cut  obtained  through 
the  strategies  in  Sec.  4.3  since  the  smooth  cut  strategy  is 
more  sensitive  to  small  shape  changes. 

4.5  Recursive  Splitting  and  Final  Merging 

Since  we  do  not  know  how  many  objects  exist  in  the  region 
for  splitting,  we  propose  a  recursive  algorithm  for  splitting 
the  region:  after  one  feasible  cut  is  found  for  the  region,  for 
the  two  new  regions  obtained  via  this  splitting,  recursively 
apply  the  splitting  algorithm  until  no  new  feasible  cut  can 
be  found.  Therefore,  multiple  cuts  can  be  found  for  one 
initial  region.  The  recursion  stops  when  no  feasible  cut 


is  found  based  on  the  strategies  in  Sec.  4.3  and  Sec.  4.4. 
In  order  to  refine  the  splitting  result,  we  apply  the  HCF 
algorithm  again  to  re-merge  regions  as  needed. 

5  Experiments 

Experiments  were  conducted  to  evaluate  the  performance 
of  the  previous  merge-only  approach  [16]  versus  the  split- 
merge  approach  described  in  this  paper.  In  the  statisti¬ 
cal  analysis,  we  used  a  test  database  of  22  leaf  images 
(about  200  leaf  objects),  20  synthesized  fish  images  (160 
fish  objects),  and  21  cell  micrographs  (about  700  cell  ob¬ 
jects).  Examples  of  these  images  are  shown  in  Figs.  4-  6. 
Deformable  template  models  were  trained  for  each  of  the 
three  shape  classes  using  additional  images  that  were  not 
included  in  the  test  database  as  described  in  [16].  Between 
40  and  70  images  were  used  to  train  each  shape  model. 

The  synthetic  images  of  fish  were  obtained  as  follows. 
Random  fish  shapes  were  created  by  drawing  samples  from 
fish  template’s  deformation  distribution  (estimated  during 
training).  The  resulting  region  was  then  partitioned  into 
sub-regions  via  triangulation  and  assigned  random  colors 
drawn  from  a  normal  distribution.  In  addition,  we  added 
random  noise  to  the  synthesized  images.  Example  syn¬ 
thetic  fish  images  are  shown  in  Fig.  4.  Fish  were  placed  at 
random  orientations  and  positions  in  each  image,  subject 
to  the  constraint  that  they  overlapped  or  touched  at  least 
one  other  fish. 

Using  the  three  databases,  we  conducted  experiments 
to  measure  the  success  rate  of  object  detection  for  the 
previous  method  (that  used  merge  only)  versus  the  new 
split/merge  approach.  The  results  of  these  experiments  are 
shown  in  Table  1.  Object  detection  rates  were  measured 
for  all  three  cut  selection  strategies  described  in  Sec.  4.3: 
short-cut  rule,  exhaustively  test  all  cuts,  and  the  queuing 
strategy.  It  was  noted  that  there  was  a  significant  improve¬ 
ment  in  object  detection  rates  when  any  of  the  splitting 
strategies  was  employed  in  concert  with  merging.  How¬ 
ever,  there  was  no  significant  difference  in  object  detection 
rates  among  the  three  different  cut  selection  strategies.  It 
was  noted  that  the  exhaustive,  test  all  cuts  strategy  may  not 
yield  the  globally  optimal  pardoning  always.  This  is  ev¬ 
idenced  by  performance  shown  for  the  cell  images  in  Ta¬ 
ble  1 .  At  each  step  in  this  strategy,  the  locally  optimal  cut 
is  selected  from  multiple  cuts. 

In  the  same  experiments,  it  was  also  noted  that  segmen¬ 
tation  quality  was  improved  when  split  and  merge  were 
employed.  As  a  quantitative  measure  for  comparing  the 
quality  of  segmentation,  the  model  fitting  cost  was  calcu¬ 
lated  for  each  region  group  using  Eq.l.  Table  2  gives  a 
comparison  of  the  mean  fitting  cost  for  deformed  shape 
models  recovered  in  the  three  different  sets  of  test  imagery. 
Smaller  mean  fitting  costs  generally  indicate  better  fitting 
models,  and  higher-quality  segmentation.  Example  results 
for  merging  versus  split/merge  are  shown  in  Figs.  4-6. 


Leaves 

Cells 

Fish 

No  splitting 

85.24% 

85.79% 

76.25% 

Short-cut  rule 

92.86% 

93.90% 

85.00% 

Test  all  cuts 

96.19% 

93.62% 

86.25% 

Queuing  strategy 

91.43% 

93.90% 

85.00% 

Table  1:  Comparison  of  object  detection  rates  in  three  test  sets. 


Leaves 

Cells 

Fish 

No  splitting 

1.1338 

1.1863 

1.0626 

Short-cut  rule 

1.1188 

1.0531 

1.0528 

Test  all  cuts 

1.1117 

1.0522 

1.0550 

Queuing  strategy 

1.1148 

1.0522 

1.0555 

Table  2:  Comparison  of  the  mean  fitting  cost  for  deformed  shape 
models  recovered  in  segmentation. 

Given  that  there  was  not  significant  difference  in  per¬ 
formance  between  the  three  splitting  strategies,  a  decid¬ 
ing  factor  in  selecting  a  strategy  would  be  CPU  time  re¬ 
quired.  In  the  experiments  it  was  noted  that  on  average 
the  short-cut  strategy  was  fastest;  on  average,  obtaining  the 
segmentation  via  short-cut  strategy  required  only  59.6%  of 
the  CPU  time  required  for  exhaustive  search.  On  average, 
the  queuing  strategy  required  79.1%  of  the  CPU  time  re¬ 
quired  for  exhaustive  search. 

One  limit  of  the  system  is  that  it  cannot  cope  with  large 
occlusions.  An  example  is  shown  in  the  second  row  of 
Fig.  6,  where  two  adjacent  cells  are  not  split  due  to  a  high 
degree  of  overlap.  Our  current  algorithm  does  not  ex¬ 
plicitly  model  occlusion.  It  processes  occluded  objects  as 
shapes  with  much  more  deformation.  Incorporation  of  an 
explicit  occlusion  model  is  needed  to  deal  with  large  over¬ 
lapping  of  shapes. 

6  Conclusion 

An  improved  method  for  deformable  shape-based  im¬ 
age  segmentation  was  described.  The  method  employs  a 
perceptually-motivated  splitting  strategy  in  concert  with  a 
deformable  template  model  that  embodies  prior  knowledge 
for  a  particular  shape  class.  The  method  combines  model 
fitting  information  in  selecting  cut  points  and  also  in  the 
evaluation  of  the  possible  cuts  in  the  region  splitting.  The 
recursive  splitting  structure  in  the  algorithm  gets  rid  of  the 
requirement  to  know  the  number  of  cuts.  This  is  an  im¬ 
provement  over  many  previous  shape  splitting  methods  and 
multi-scale  techniques [24,  15,  1,  21].  The  experimental 
results  show  that  after  the  splitting,  the  success  rate  of  ob¬ 
ject  detection  and  the  accuracy  of  object  boundaries  are 
improved.  Some  different  strategies  are  tested  in  our  ex¬ 
periments  for  the  feasible  cut  selection.  When  considering 
CPU  time  employed,  the  modified  short-cut  rule  is  faster 
in  searching  feasible  cuts,  and  the  splitting  quality  is  com¬ 
parable  to  other  strategies. 
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Figure  4:  Region  merging  and  split/merge  experiment  with  a 
synthesized  fish  image,  (a)  Original  image,  (b)  initial  over¬ 
segmentation  result,  (c)  merging  result  based  HCF  algorithm,  (d) 
overlapping  between  the  recovered  models  (thick  black  contours) 
and  the  regions  (grey  areas),  (e)  result  of  merging  again  after  split¬ 
ting,  (f)  new  recovered  models. 
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Figure  5:  Two  examples  from  the  leaf  image  database:  (a)  original  image,  (b)  over- segmentation,  (c)  merging  result  after  HCF  algorithm, 
showing  overlap  between  the  recovered  models  (thick  black  contours)  and  the  regions  (grey  areas), (d)  model-based  splitting  result. 
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Figure  6:  Example  image  from  blood  cell  micrograph  database:  (a)  original  image,  (b)  over-segmentation  result,  (c)  merging  result 
based  HCF  algorithm  showing  overlap  between  the  recovered  models  (thick  black  contours)  and  the  regions  (grey  areas),  (d)  model- 
based  splitting  result. 


